Generation of mixed media streams

ABSTRACT

To achieve coordination during generation of a plurality of mixed media streams, there is suggested a method of generating a mixed media stream ( 62 - 64 ) from input media streams ( 40 - 50 ) having payload data elements and related identifiers. The method comprises the step (S 12 ) of aligning the input media streams ( 40 - 50 ) according to a pre-specified relation between identifiers in different input media streams ( 40 - 50 ) before generating the mixed media stream.

FIELD OF INVENTION

The present invention relates to the generation of mixed media streamsfrom input media streams having payload data elements and relatedidentifiers, and in particular considering a relation between differentmedia streams during generation of a mixed media stream.

BACKGROUND ART

Mixed media streams are generated in communication networks, e.g., whenmixing audio signals and video signals during a video conference. Here,it is important that mixed audio signals are matched to the relatedmixed video signal because otherwise the speech will not belip-synchronous to the video stream. The same problem also arises withother streams than audio or video streams, e.g., text streams havingalphanumeric signs when using subtitles.

Currently, the matching of such video and audio or in more general sensemixed media streams require a complicated procedure. Normally, timestamps are used, which will be attached to the different signals toenable the matching of the related media streams. However, while this atleast is some mechanism to a match the mixing of different mediastreams, currently, there does not exist any solution to the problem howthe generation of mixed media streams of a different type, e.g., a mixedvideo stream and a mixed audio stream, may be coordinated.

SUMMARY OF INVENTION

In view of the above, the object of the present invention is to providea mechanism for coordination during generation of a plurality of mixedmedia streams.

According to the present invention, this object is achieved through amethod of generating a mixed media stream from input media streamshaving payload data elements and related identifiers, wherein the inputmedia streams are aligned according to a pre-specified relation betweenidentifiers in different input media streams before generating the mixedmedia stream.

In the sense of the present invention a media stream is generated in apacket switched transmission process typically by a user end equipment.One example of an input media stream may be a voice data packet streamor a video data packet stream which is generated in a video conference.Another example would be the exchange of media streams over any type ofnetwork, e.g., the Internet, an Intranet, an LAN, a WAN or VPN.

Typically, each media stream consists of a plurality of data packets andeach data packet has a header part and a payload data part. The headerpart carries identifiers characterizing a time relationship betweendifferent data packets in a single media stream. A single end userequipment generates different media streams—e.g., voice, data, text,sound etc.—and data packets of different type media streams carry thesame identifiers when they are generated at the same instance of time.

When different end user equipments generate different media streams ofcomparable type each such media streams for each end user equipment willthen be split between different mixers in the communication systemshandling the data packet exchange before mixing of similar type mediastreams into a mixed media stream and subsequent transmission over thecommunications network.

The present invention considers the coordination of the operation fordifferent mixers.

To achieve this, it is proposed to add a further step of alignment ofdifferent input media streams in time before generating the mixed mediastream. The alignment in time is achieved such that a pre-specifiedrelative relation in time between different media streams is achieved.

Here, use is made of identifiers being available in the input mediastreams. As outlined above, in each media stream there are availableidentifiers defining an order of data packets in that media stream. Thesame also applies for all other input media streams which will finallybe used to generate a specific mixed media stream.

Therefore, at a certain point in time one may consider a tupel of suchidentifiers in each input media stream as a relation which may then becompared to a pre-specified relation.

In other words, each such tupel defines a relative alignment of inputmedia streams which may be changed according to a pre-specified relationeither through advancing or delaying input media streams in time.

Therefore, the result of the inventive method is a modification of therelative alignment of input media streams according to a pre-specifiedrelation of ordering for the input media streams before generating themixed media streams. It should be noted that such an alignment isrelated to the relative alignment of input media streams only and doesnot rely on absolute time.

According to a preferred embodiment of the present invention thepre-specified relation between identifiers in different input mediastreams is matched to a relation between identifiers in further inputmedia streams used during generation of a further mixed media stream.

Here, one example could be that the mixing of a first group of inputmedia streams is related to speech data and the mixing of a second groupof input media streams is related to video, text, service-related data,etc. Then, what is assured is that the generated first mixed mediastream and the generated second mixed media will be generated using thesame relative relation between identifiers in the related groups ofinput media streams. In other words, the generated mixed media streamsare generated with the related input media streams being alignedaccording to the same relative relation between identifiers.

Here, a relation between identifiers in a first mixer—that may also bereferred to a master mixer, without limiting scope of invention—may beused for the generation of a further mixed media stream in a secondmixer—which may be referred to as slave mixer, without limiting scope ofinvention.

It should be noted that according to the present invention each type ofmixer in the communication network may be operated either as a mastermixer or a slave mixer as long as it implements the inventive method.

It should further be noted that according to the present invention afurther alternative is that the relation between identifiers from inputmedia streams may be determined in a stand-alone alignment unit providedexternally to mixers adapted to generate mixed media streams. Here, thealignment unit would inform the mixers about the relation betweenidentifiers for subsequent processing of input media streams in themixers.

According to a further preferred embodiment of the present invention, amatching of relations between identifiers in a first group of inputmedia streams and in a further group of input media streams is achievedin an efficient way.

Initially, an intersection between the input media streams and thefurther input media streams is identified. The purpose of this step isto tackle situations where not every mixer is operating on the same andrelated number of input media streams.

One such situation may, e.g., occur if different user end equipmentsparticipating in a communication session generate different mediastreams, e.g., a first user end equipment may generate voice data only,a second user end equipment may generate video data only, and a thirdand fourth user end equipment may generate both voice and video data. Inthis case, only a coordination of the mixing for media streams of thethird and fourth user end equipment may make sense as only here a mutualrelative relation between identifiers fixing a relative order datapackets in a media stream is available.

According to the preferred embodiment, once such an intersection isidentified what follows is the determination of the relation betweenidentifiers in input media streams at a master mixer for subsequent useduring mixing of input media streams in a slave mixer.

In a further step, those input media streams in the slave mixer whichare comprised in the identified intersection will be aligned accordingto the relation of identifiers of the input media streams handled by themaster mixer.

According to another preferred embodiment of the present invention themethod is efficiently implemented through use of a set representation ofinput media streams. Each group input media stream is characterized by aset of tupels. A first element in each tupel characterizes an inputmedia stream, e.g., through an input media stream number, and a secondelement in each tupel defines an identifier of the input media stream ata point in time. Thus, the determination of the intersection explainedabove may be achieved through simple set operations.

A further preferred embodiment of the present invention is particularlysuited to a situation where related input media streams in differentgroups of input media streams are not supplied to corresponding inputterminals of different mixers.

Here, one example would be that a first user end equipment generates afirst voice media stream and a first media stream and a second user endequipment generates a second voice media stream and a second video mediastream. Nevertheless, a situation may occur where at a first mixerhandling voice media streams the first and second voice media streamsare supplied to the first and second input terminal, while at a secondmixer handling video media streams the first video media stream and thesecond video media stream are supplied to the second and first inputterminal, respectively, or in other words, are interchanged.

In this case, it is very useful to re-order the sequence of input mediastreams and the input terminals either at the master mixer or at theslave mixer because otherwise the alignment procedure outlined abovewill lead to a non-intended result. Preferably, such a re-ordering maybe described using a permutation vector which easily allows to extendthe set operations described above for incorporation of the re-orderinginto the processing of different groups of input media streams.

Another preferred embodiment of the present invention relates to the waythe alignment of input media streams is achieved.

An important aspect of the alignment is the determination of a timedelay for each input stream such that—assuming that the time delay issubsequently applied to the input media stream—the relations between theidentifiers in the input media streams then correspond to thepre-specified relation of identifiers.

Also, the time delay may be achieved through shifting each input mediastream according to the related time delay, e.g., using a buffer memoryof suitable size.

Yet another preferred embodiment of the present invention is related tothe exchange of information regarding the pre-specified relation betweenidentifiers of different input media streams.

A first way to achieve this exchange is through forwarding a relatedsignal from the master mixer to the slave mixer which signal shouldcomprise at least the relation of identifiers used during generation ofa mixed media stream in the master mixer.

Another mechanism suitable for the exchange of tupel between a mastermixer and a slave mixer may be a shared memory provided between a mastermixer and the slave mixer, where each mixer has access to the sharedmemory in a time coordinated manner.

Similar advantages as outlined above with respect to the inventivemethod are also achieved by an inventive apparatus for generating amixed media stream from input media streams having payload data elementsand related identifiers and preferred embodiments thereof.

Further, according to yet another preferred embodiment of the presentinvention there is provided a computer program product directly loadableinto the internal memory of a media mixer comprising software codeportions for performing the inventive mixing process when the product isrun on the processor of the media stream processor.

Therefore, the present invention is also provided to achieve animplementation of the inventive method steps on computer or processorsystems. In conclusion, such implementation leads to the provision ofcomputer program products for use with the computer system or morespecifically a processor comprised in, e.g., a media stream mixer.

This programs defining the functions of the present invention can bedelivered to a computer/processor in many forms, including, but notlimited to information permanently stored on non-writable storage media,e.g., read only memory devices such as ROM or CD ROM discs readable byprocessors or computer I/O attachments; information stored on writablestorage media, i.e. floppy discs and harddrives; or information conveyto a computer/processor through communication media such as networkand/or telephone networks and/or Internet via modems or other interfacedevices. It should be understood that such media, when carryingprocessor readable instructions implementing the inventive conceptrepresent alternate embodiments of the present invention.

DESCRIPTION OF DRAWINGS

In the following preferred embodiments of the present invention will bedescribed with reference to the drawings in which:

FIG. 1 shows a schematic diagram of a media stream mixer according tothe present invention;

FIG. 2 shows a flowchart illustrating the operation of the media streammixer according to the present invention;

FIG. 3 shows a schematic diagram of the alignment unit shown in FIG. 1;

FIG. 4 shows a flowchart illustrating the operation of the alignmentunit shown in FIG. 3;

FIG. 5 shows a first example for the alignment of media streamsaccording to the present invention;

FIG. 6 shows a second example for the alignment of media streamsaccording to the present invention; and

FIG. 7 shows a third example for the alignment of media streamsaccording to the present invention.

DESCRIPTION OF BEST MODE AND PREFERRED EMBODIMENTS OF THE INVENTION

In the following, general reference will be made to the mixing of mediastreams. Here, it is assumed that each media stream is set up by asequence of data packets dividing into a header part and a payload part.The header part will comprise identifiers defining a sequential orderingof the data packets. Media stream mixing in the sense of the presentinvention is related to the reception of a plurality of media streams ofa common type and the processing of the plurality of media streams suchthat only a single media stream of a mixed type according to a specificprocessing on the input media streams is output. The present inventionis well suited for any type of processing on input media streams, e.g.,audio streams, video streams, and/or text streams. In other words, themixing of media streams is not restricted to any type of media streams,e.g., voice, data, text, etc.

FIG. 1 shows a schematic diagram of a media stream mixer according tothe present invention.

As shown in FIG. 1, the media stream mixer 10 comprises a stream inputunit 12 adapted to receive at least one input media stream, a streamprocessing unit 14 generating a single output media stream from the atleast one input media stream, and a stream output unit 16 for output ofthe generated mixed media stream.

Further, the media stream mixer 10 comprises a memory 18 and aninterface unit 20 for exchange of identifier information to be used inthe stream processing unit. The stream processing unit 14 comprises analignment unit 22, a mixer unit 24, and an identifier unit 26. While inFIG. 1 the alignment unit is shown in relation to the stream processingunit 14, it should be noted that according to the present invention itis also possible to provide the alignment unit externally to the streamprocessing unit 14, e.g., in the stream input unit 12 or as stand-aloneunit outside the media stream mixer.

FIG. 2 shows a flowchart illustrating the operation of the media streammixer according to the present invention.

As shown in FIG. 2, in a first operation step S10 the media stream mixerwill receive at least one input media stream of a pre-specified type.Then, in a step S12 the media stream mixer will align input mediastreams according to a pre-specified relation, and in a step S14 themedia stream mixer will output the aligned media stream.

Heretofore, the stream processing unit 14 of the media stream mixer willexchange identifiers defining a pre-specified relation to be achievedfor the input media stream before mixing thereof, by the identifierinterface 20.

For comparison of a pre-specified relation of identifiers with theactual relation of identifiers in different input media streams, thestream processing unit 14 comprises the identifier unit 26 adapted toextract identifiers from input media streams.

The step of alignment S12 shown in FIG. 2 is achieved by the alignmentunit 22 shown in FIG. 1, and the mixing of the input media streams ofalignment is achieved by the mixer unit 24.

In the following, a more detailed explanation of the alignment of inputmedia streams according to the present invention will be given withrespect to FIGS. 3 and 4.

FIG. 3 shows a schematic diagram of the alignment unit 22 shown inFIG. 1. As shown in FIG. 3, the alignment unit comprises apre-processing unit 28, a stream shift calculation unit 30 and a streamshift unit 32.

Operatively, the pre-processing unit 28 is adapted to identify whichinput media streams of the plurality of input media streams must bealigned according to a pre-specified relation.

The stream shift calculation unit 30 is adapted to determine a correctordering of the sequence of input media streams. Further, once relevantinput media streams are brought into a proper order, or in other words,into the same order as input media streams considered for defining thepre-specified relation between identifiers available in the mastermixer—the stream shift calculation unit 30 will then proceed with thecalculation of an appropriate shift to achieve the required alignment ofinput media streams in the stream shifting unit 32.

Finally, the stream shifting unit 32 will achieve an alignment of inputmedia streams according to the result determined by the stream shiftcalculation unit 30, e.g., through intermediate storage of specificinput media streams in the memory 18 of the media stream mixer 10.

The operation of the alignment unit shown in FIG. 3 may be summarizedaccording to a flowchart illustrating the operation as shown in FIG. 4.

As shown in FIG. 4 the alignment unit 22 basically identifies whichinput media streams to the mixed media mixer must be processed in a stepS20. Then, it determines an appropriate time delay with those mediastreams which have to be processed in a step S22. In a final step S22 itwill shift media streams to be processed according to the determinedtime delay in a step S24, before subsequent mixing thereof.

In the following, the operation of the media stream mixer illustratedwith respect to FIG. 1 to 4 will be described on the basis of a moreformal notion.

Heretofore, it may be assumed that a first mixer is considered as amaster mixer indicated with index_(M) and a second mixer is consideredas slave mixer indicated with index_(S).

The set of media streams handled by the master mixer may be describedthrough a set of tupel s_(i), i_(i), where s_(i) relates to a number ofeach input media stream and i_(i) relates to an identifier of thisstream s_(i) at a point in time which changes during operation on theset of input streams.

Similarly, for the slave mixer the set of input media streams at a pointin time is described by a set of tupel the q_(i) is related to thenumber of a specific input media stream and related n_(i) is theidentifier comprised in the input media stream at a point in time:I_(M)=[(s₁,i₁),(s₂,i₂), . . . ,(s_(m),i_(m))]I_(s)=[(q₁,n₁),(q₂,n₂), . . . ,(q_(s),n_(s),)]

Here, one example may be that a master mixer operates on a set of inputmedia streams 2, 4, 5, which at a point in time carry identifiers 1, 4,and 7. Further, for illustrating purposes one may assume that the slavemixer operates on a set of input media streams 2, 5, 4, 10 which at apoint in time carry the identifiers 2, 1, 5, 2.I_(M)=[(2,1),(4,4),(5,7)]I_(s)=[(2,2),(5,1), (4,5),(10,2)]

In view of the examples given above, the present invention now considershow the ordering and time relationship between the different input mediastreams at the slave mixer may be modified such that they correspond tothe ordering the time relationship between the input media streams andthe master mixer.

Here, it should be clear that a modification of input media streams isonly achievable in the slave mixer for those input media streams whichhave a counter part in the master mixer. Therefore, a first step is thedetermination of a subset of input media streams at the master mixer andslave mixer which are coincident according toS_(A)={s₁,s₂, . . . ,s_(m)}∩{q₁,q₂, . . . ,q_(s)}

For the particular example given above the result would be the inputmedia streams having the number 2, 4, 5, respectively:S_(A)={2,4,5}

In view of the determined intersection of input media streams at themaster mixer and the slave mixer, one may then modify the set of inputmedia streams at the master mixer and the slave mixer to be incompliance with the intersection according to$I_{M}^{\prime} = {\bigcup\limits_{i \in {\lbrack{1,\ldots\quad,m}\rbrack}}\{ {{( {s_{i},i_{i}} )\text{❘}s_{i}} \in S_{A}} \}}$$I_{S}^{\prime} = {\bigcup\limits_{j \in {\lbrack{1,\ldots\quad,s}\rbrack}}\{ {{( {q_{j},n_{j}} )\text{❘}q_{j}} \in S_{A}} \}}$which for example leads toI_(M) ^(′)=[(2,1),(4,4),(5,7)]I_(s) ^(′)=(2,2),(5,1),(4,5)]

From the example one may see that not only the sequence of the inputmedia streams must be considered, but also their ordering. This orderingis—according to the stream numbers—2, 4, and 5 for the master mixer, but2, 5, 4 for the slave mixer. This implies that it is not possible tocarry out an immediate consideration of identifiers of the input mediastreams at the master mixer and the slave mixer without appropriatemodification of the input media stream ordering.

Such a modification may be represented by a permutation vector. Eachelement of the permutation vector expresses which input media stream tothe master mixer corresponds to the which input media stream of theslave mixer according to$\underset{i \in {\lbrack{1,\ldots\quad,{I_{S}^{\prime}}}\rbrack}}{\forall}\{ {{p(i)} = {{j \in {\lbrack {1,\ldots\quad,{❘{I_{S}^{\prime}  \rbrack}}} s_{i}}} = q_{j}}} \}$

For the specific example referred to above, the result isp=[1,3,2]

The different steps described so far in a formal manner and also in viewof an example may be achieved in the pre-processing unit 28 shown inFIG. 3.

The further steps to be explained in the following are achieved by thestream shift calculation unit 30 also shown in this FIG. 3.

The first step to be achieved by the stream shift calculation unit 30 isto modify the set of input streams to the slave mixer according to thepermutation vector derived as outlined above. In other words, the set ofinput media streams to the slave mixer is brought into an orderingaccording to the set of input media streams to the master mixer:I_(s) ^(″)=ØI_(s) ^(″=I) _(s) ^(″)∪(q_(p(i)),n_(p(i))),i=1, . . . ,|I_(s) ^(′)|which for the example referred to above leads toI_(s) ^(″)=[(2,2),(4,5),(5,1)]

One should note that the modification of the set of input streams to theslave mixer must not be implemented through actual re-ordering of inputmedia streams at the input terminals of the slave mixer but may beachieved through indirect access to the input media streams using thepermutation vector, as will be explained in the following.

Operatively, the stream shift calculation unit 30 determines eachidentifier in the input media streams to the master mixer and comparesit with the related identifiers of input media streams to the slavemixer. The comparison is based onδ₁=i₁−n_(p(1)),i=1, . . . ,|I_(s) ^(′)|

Here, the selection of the write input media stream at the slave mixerfor comparison with the related input media stream to the master mixeris achieved through indirect access to the identifiers of the inputmedia streams to the slave mixer using the permutation vector. For thespecific examples given above, the result of this step will beδ=[−1,−1,6]

As can be seen from the above, there may be input media streams to theslave mixer which are either delayed or advanced in relation to thecorresponding input media stream to the master mixer.

However, as the stream shift unit 32, explained in more detail in thefollowing, may only delay input media streams at the slave mixer, onehas to take care that the final alignment consists solely of delays ofinput media streams to the slave mixer.

According to the present invention, this is no problem as only arelative relation between input media streams must be maintained at theslave mixer in view of a relation of identifiers in input media streamsat the master mixer without an absolute time scale.

Therefore, one will modify the delay vector generated through comparisonof identifiers in the input media streams to the master mixer and theslave mixer according to${\delta_{i}^{\prime} = {\delta_{i} - {\max\limits_{j \in {\lbrack{1,\ldots\quad,{I_{S}^{\prime}}}\rbrack}}\delta_{j}}}},{i = 1},\ldots\quad,{I_{S}^{\prime}}$

For the example referred to above, this leads toδ^(′)=[−7,−7,0]

As can be seen, after this step only delays for the input media streamsto the slave mixer occur. These delays are realized through intermediatestorage of data packets comprised in each of the input media streams tothe slave mixer.

In other words, the delay vector generated in the stream shiftcalculation unit 30 will be used by the stream shifting unit 32 suchthat the input media streams to the slave mixer are delayed according toq_(s) ₁ ^(′)(t)=q_(s) ₁ (t+δ₁ ^(′))which for the example referred to above leads toq₂ ^(′)(t)=q₂(t−7)q₄ ^(′)(t)=q₄(t−7)q₅ ^(′)(t)=q₅(t)

It should be noted that the formulas specified above may be modified inreversing the signs and the maximum operator to the minimum operatorwhich would lead to the same result.

FIG. 5 shows a first example for the alignment of media streamsaccording to the present invention.

As shown in FIG. 5, according to a first example, three user endterminals 34, 36, and 38 generate simultaneously two types of mediastreams 40, 42, 44, 46, and 48, 50. Here, according to the example shownin FIG. 5, the media streams 40, 44, and 48 are of similar type, andalso the media stream 42, 46, 50 may be of similar type. Similar mediastreams are supplied to a related mixer via splitters 52, 54, and 56.

In more detail, these splitters 52, 54, 56 submit the media streams 40,44, 48 to a master mixer and the media streams 42, 46, 50 to a slavemixer 60. As outlined above, each master mixer 58 and slave mixer 60generates a single output media stream 62 and 64 which may then beforwarded to the receiving equipment 66.

It should be noted that each such generated output media stream and therelated data packets comprised therein are identified by a master ID.

According to the present invention, the relation between identifiers ininput media streams at the master mixer 58—e.g., i6, i22, i13, and themaster ID will then be forwarded at specific points in time—eitherperiodically or not—to the slave mixer 60 via a signalling line.Although not shown in FIG. 5, an alternative mechanism for data exchangebetween the master mixer 58 and the slave mixer 60 may be a sharedmemory type communication.

FIG. 6 shows a second example of the alignment of media streamsaccording to the present invention.

The example shown in FIG. 6 is similar to the example shown in FIG. 6,so that the same reference numerals are used to denote the samestructural elements. In the example shown in FIG. 6, data packets ofinput media streams and output media streams are shown for one point intime.

According to the example shown in FIG. 6, each end user equipment 34,36, 38 generates a video media stream and an audio media stream whichare distributed via splitters 52, 54, 56 to a video mixer 58 operatingas master mixer and an audio mixer operating as slave mixer.

The time relationship between the video input media streams and theirrelated data packets at the point in time shown in FIG. 6 is (12, 23,42) and is exchanged from the video mixer 58 to the audio mixer 60.

Further, the audio mixer will receive the master. ID=1 of the datapacket generated by the video mixer 58 for forwarding to the receivingequipment 66. The audio mixer 60 will receive the information from thevideo mixer and combine data packets having identifiers (12, 23, 42)also for the audio input media streams and generate a related outputdata packet under the same master ID.

Therefore, at the audio mixer not only the relative time relationshipbetween data packets in video input media stream used at the video mixeris considered for generation of an output data packet of the audio type,but this output data packet is also characterized by the same master ID,so that at the receiving equipment 66, both the mixed video data packetand the mixed audio data packet are referenced under the correct masterID. Therefore, video display and audio display will be in proper timerelationship.

FIG. 7 shows a third example of the alignment of the media streamsaccording to the present invention.

The example shown in FIG. 7 is different over the previous examples inthat the number of input media streams handled by the master mixer andthe slave mixer is different. Further, the input media streams at thesecond and third input terminal of the master mixer and the slave mixerare interchanged. Finally, for this example it is assumed that also themutual relation between input media streams at the master mixer isdifferent from that of the slave mixer.

As shown in FIG. 7, to achieve the same relative relation betweenidentifiers (1, 4, 7), also at the slave mixer it will be necessary toapply an intermediate storage for the input media streams supplied toinput terminal q2 of the slave mixer. In other words, as shown on theright side of FIG. 7, at the slave mixer at input terminal 1 and 3,there are supplied input media streams 2, 4 according the input mediastreams supplied to the first and second input terminals of the mastermixer.

Furthers the difference between the related identifiers (5−2=3) issimilar to those of the related input media streams at the master mixer(4−1 =3).

However, the input media stream supplied to the second input terminal ofthe slave mixer, i.e. input media stream No. 5 corresponding to theinput media stream supplied to the third input terminal of the mastermixer is not aligned properly.

To achieve the correct alignment, as shown in the lower table of FIG. 7,the input media stream to the first and third terminal of the slavemixer—i.e. the input media stream with Nos. 2 and 4—should be suppliedto an intermediate storage, while the processing of the input mediastream to the second input terminal of the slave mixer is continuedwithout delay.

After processing of seven such data packets being related to the inputmedia stream to the second input terminal of the slave mixer, therelated identifier in the input data packet will be (8). The differencebetween the input media stream being supplied to the second inputterminal of the slave mixer and the input media stream being supplied tothe third input terminal of the slave mixer will be (8−5=3). Thisdifference is in compliance with the difference between identifiers inthe input media streams supplied to the second and third input terminalof the master mixer (7−4=3).

At this instance of time, it is then possible to combine the buffereddata packets of the input media streams with Nos. 2 and 4 and suppliedto the first and third input terminal of the slave mixer with the inputmedia streams being supplied to the second input terminal of the slavemixer and being processed without buffering, as shown with the shadedrectangles in FIG. 7.

According to the present invention, it is also suggested as option tomaintain a relation of the input media stream supplied to the fourthinput terminal of the slave mixer during subsequent processing of allinput media streams to the slave mixer.

While above reference has been made to identifiers in input mediastreams, generally, examples of such identifiers could be time stamps orsequence numbers. Also, the exchange of a relation such identifiers fromthe master mixer to the slave mixer may be achieved periodically whensending a SYNC signal to slave mixers. Here, one may either send allidentifiers being handled in the master mixer or information about anabsolute identifier value and the relation of identifiers, e.g., ID1=x,ID2=ID1+y, ID3=ID+z, etc.

Further, the data exchange between the master mixer and the slave mixerhas almost no delay requirement and the periocity may not be very high,assuming a reasonable low drift between the clock sources in thedifferent mixers. The exchange signal, e.g., the SYNC signal, may besent in different ways, as timing requirements are not very high.Example implementations would be out-band signalling, in-band signallingor via an operating system.

1. A method of generating a mixed media stream from input media streamsof a first type having payload data elements and related identifiers,respectively, comprising the step: aligning the input media streams ofthe first type according to a pre-specified relation between identifiersin the input media streams of the first type before generating the mixedmedia stream, wherein the pre-specified relation between identifiers inthe input media streams of the first type is matched to a relationbetween identifiers in further input media streams of a second type usedduring generation of a further mixed media stream from the input mediastreams of the second type.
 2. The method according to claim 1, whereinthe matching of relations between identifiers in the input media streamsof the first type and further input media streams of the second type isachieved by identifying an intersection between the input media streamsof the first type and the further input media streams of the secondtype, determining a relation between identifiers in the further inputmedia streams of the second type for those further input media streamswhich are comprised in the intersection, and aligning the input mediastreams of the first type which are comprised in the intersectionaccording to the relation of identifiers in the further input mediastreams of the second type.
 3. The method according to claim 2,characterized in that the input media streams of the first type at apoint in time are described by I_(s)=[(q₁,n₁),(q₂,n₂), . . .,(q_(s),n_(s))] with {q₁,q₂, . . . , q_(m)} as a set of input mediastreams and {n₁,n₂, . . . ,n_(s)} as a set of identifiers in the inputmedia streams at a point in time, the further input media streams of thesecond type at a point in time are described by I_(M)=[(s₁,i₁),(s₂,i₂),. . . ,(s_(m),i_(m))] with {s₁,s₂, . . . ,s_(m)} as a set of furtherinput media streams and {i₁,i₂, . . . i_(m)} as a set of identifiers inthe further input media streams of the second type at the point in time,the intersection between the input media streams of the first type andthe further input media streams of the second type is S_(A)={s₁,s₂, . .. ,s_(m)}∩{q₁,q₂, . . . q_(s)}, and the relation between identifiers inthe further input media streams of the second type for those furtherinput media streams of the second type which are comprised in theintersection is determined by$I_{S}^{\prime} = {\bigcup\limits_{j \in {\lbrack{1,\ldots\quad,s}\rbrack}}{\{ {{( {q_{j},n_{j}} )\text{❘}q_{j}} \in S_{A}} \}.}}$4. The method according to claim 3, further comprising the step ofre-ordering the sequence of input media streams of the first type in theintersection according to the sequence of further input media streams ofthe second type in the intersection.
 5. The method according to claim 4,wherein the re-ordering of the sequence of input media streams of thefirst type is achieved according to a permutation vector defined by$\underset{i \in {\lbrack{1,\ldots\quad,{I_{s}^{\prime}}}\rbrack}}{\forall}\{ {{p(i)} = {j \in {\lbrack {1,\ldots\quad,{{{I_{s}^{\prime}}s_{i}} = q_{j}}} \}.}}} $6. The method according to claim 2 wherein the alignment of input mediastreams of the first type in the intersection is achieved by determininga relative time delay for each input media stream of the first type suchthat relations between different identifiers in the input media streamsof the first type after time delay correspond to relations betweenidentifiers in the further input media streams of the second type, andshifting each input media stream of the first type in time according tothe related time delay.
 7. The method according to claim 6, wherein thetime delay for each input media stream of the first type is determinedaccording to δ_(i) = i_(i) − n_(p(i),)i = 1, …  , I_(s)^(′)${\delta_{i}^{\prime} = {\delta_{i} - {\max\limits_{j \in {\lbrack{{1\ldots}\quad,{I_{s}^{\prime}}}\rbrack}}\delta_{j}}}},{i = 1},\ldots\quad,{I_{s}^{\prime}}$and each input media stream of the first type is shifted in timeaccording toq_(s) _(i) ^(′)(t)=q_(s) _(i) (t+δ_(i) ^(′)),i=1, . . . , |I_(s) ^(′)|.8. The method according to claim 1 wherein, an exchange of informationregarding the pre-specified relation between identifiers is achievedthrough a signal transfer or a shared-memory mechanism.
 9. An apparatusfor generating a mixed media stream from input media streams of a firsttype having payload data elements and related identifiers of a secondtype, comprising: an identifier interface unit for exchange of apre-specified relation between identifiers in different input mediastreams, and an alignment unit adapted to align the input media streamsof the first type according to the pre-specified relation betweenidentifiers in different input media streams of a second type beforegenerating the mixed media stream.
 10. The apparatus according to claim9, wherein the pre-specified relation is matched to a relation betweenidentifiers in further input media streams of the second type usedduring generation of a further mixed media stream and that the matchingof relations between identifiers in the input media streams of the firsttype and in the further input media streams of the second type isachieved by a pre-processing unit adapted to identify an intersectionbetween the input media streams of the first type and the further inputmedia streams of the second type, a calculation unit adapted todetermine a relation between identifiers in the further input mediastreams of the second type for those further input media streams of thesecond type which are comprised in the intersection, and an aligningunit adapted to align the input media streams of the first type whichare comprised in the intersection according to the relation ofidentifiers in the further input media streams of the second type. 11.The apparatus according to claim 10, wherein the input media streams ofthe first type at a point in time are described byI_(s)=[(q₁,n₁),q₂,n₂), . . . ,(q_(s),n_(s))] with {q₁,q₂, . . . ,q_(m)}as the set of input media streams of the first type and {n₁,n₂, . . .,n_(s)} as a set of identifiers in the input media streams of the firsttype at a point in time, the further input media streams of the secondtype at a point in time are described by I_(M)=[s₁,i₁),(s₂,i₂), . . .,(s_(m),i_(m))] with {s₁,s₂, . . . ,s_(m)} as a set of further inputmedia streams of the second type and {i₁,i₂, . . .,i_(m)} as a set ofidentifiers in the further input media streams of the second type at thepoint in time, the pre-processing unit is adapted to determine theintersection between the input media streams of the first type and thefurther input media streams of the second type according toS_(A)={s₁,s₂, . . . , s_(m)}∩{q₁,q₂, . . . ,q_(s)}, and thepre-processing unit is further adapted to determine the relation betweenidentifiers in the further input media streams of the second type forthose further input media streams of the first type which are comprisedin the intersection on the basis of$I_{S}^{\prime} = {\bigcup\limits_{j \in {\lbrack{1,\ldots\quad,s}\rbrack}}{\{ {{( {q_{j},n_{j}} )\text{❘}q_{j}} \in S_{A}} \}.}}$12. The apparatus according to claim 10 wherein the pre-processing unitis further adapted to re-order the sequence of input media streams ofthe first type in the intersection according to the sequence of furtherinput media streams of the second type in the intersection.
 13. Theapparatus according to claim 12, wherein the pre-processing unit isadapted to re-order the sequence of input media streams of the firsttype according to a permutation vector defined by$\underset{i \in {\lbrack{{1\ldots}\quad,{I_{S}^{\prime}}}\rbrack}}{\forall}{\{ {{p(i)} = {{j \in {\lbrack {1,\ldots\quad,{❘{I_{s}^{\prime}  \rbrack}}} s_{i}}} = q_{j}}} \}.}$14. The apparatus according to claim 9, wherein the alignment of inputmedia streams of the first type in the intersection is achieved by thecalculation unit adapted to determine a relative time delay for eachinput media stream of the first type such that relations betweendifferent identifiers in the input media steams of the first types aftertime delays correspond to relations between identifiers in the furtherinput media streams of the second type, and a shifting unit adapted toshift each input media stream of the first type in time according to therelated time delay.
 15. The apparatus according to claim 14, thecalculation unit is adapted to calculate the time delay for each inputmedia stream of the first type according toδ_(i) = i_(i) − n_(p(i),)i = 1, …  , I_(s)^(′)${\delta_{i}^{\prime} = {\delta_{i} - {\max\limits_{j \in {\lbrack{{1\ldots}\quad,{I_{s}^{\prime}}}\rbrack}}\delta_{j}}}},{i = 1},\ldots\quad,{I_{s}^{\prime}}$and the shifting unit is adapted to shift each input media stream of thefirst type in time according toq_(s) _(i) ⁴⁰(t)=q_(s) _(i) (t+δ_(i) ^(′)),i=1, . . . ,|I_(s) ^(′)| 16.(canceled)